® 




Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



Publication number: 



0 350 149 

A2 



® 



EUROPEAN PATENT APPLICATION 



(g) Application number: 89304046.9 


0 int CI/. H04J 3/06 


@ Date of filing: 24.04.89 




(x) Priority: 30-06.88 US 213746 


Vjy Applicant: DliallAU tuiuirivicni 
CORPORATION 


® Date of publication of application: 
10.01.90 Bulletin 90/02 


146 Main Street 
Maynard, MA 01754(US) 


® Designated Contracting States: 
DE FR G6 IT NL 


@ Inventor: Fine, Michael 
1 Ballord Place 
Cambridge, MA 07139(US) 

© Representative: Goodman, Christopher et al 
Eric Potter & Clarlison 14 Oxford Street 
Nottingham NQ1 5BP(GB) 



® Method of maintaining a correct time In a distributed processing system. 

© A method for maintaining a correct time in a distributed processing system involves clerk nodes maintaining 
their local clocks by requesting time intervals from server nodes, and server nodes maintaining their local clocks 
either by requesting time Intervals from other server . nodes or by receiving time information from an outside 
source. Server nodes also provide time intervals to requesting clerk nodes and requesting server nodes with 
such a method. System time is always increasing and monotonic and faulty servers are detected penodically. 
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BACKGROUND OF THE INVENTION 



The present invention relates to distributed processing systems, and specifically to a method of 
maintaining a correct time in a plurality of nodes of a distributed processing system. 



5 a) GENERAL NETWORK OVERVIEW 

Some distributed data processing systems are configured as Local Area Networks (LAN's) having 
multiple nodes connected together for passing information back and forth. One such distributed data 
processing system network 29 is shown in Fig. 1. The network consists of multiple nodes 30 which pass 
10 information along connecting lines 32. The connecting lines 32 may be twisted-pair wires, coaxial cables, or 
the like. The nodes 30 may be any of various types of devices hooked to the network, such as VAXclusters 
provided by Digital Equipment Corporation, microcomputers, intelligent workstations, mainframe computers 
or the like. 

All nodes in the network 29 contain software responsible for organizing and coordinating the flow of 

75 information between the nodes. One type of such software is dedicated to the mechanics of establishing a 
signal connection between nodes, of transmitting information between nodes, and of ending the connection 
when the information transmittal is complete. Such software operates according to one of a variety of 
communication protocols, such as Digital Network Architecture (DNA) or Open System Interconnection 
(OSI). which are well known in the art and will not be described herein. 

20 A second type of software in the nodes is dedicated, not to the mechanics of information transmittal, 
but to generating the information being transmitted, i.e., to application programs. Examples of application 
programs include word processors, mail programs, data base managers, etc. 

A third type of software in the nodes provides an easy-to-use interface between the application 
programs and various other hardware or software. Software programs of this third type are called "clerks," 

25 "clerk software." or "clerk programs" because they handle the details of information transfer for the 
application programs. Application programs interfacing with clerks are called "clients." "client software." or 
"client programs" because the application programs are being served by the clerks. Nodes containing clerk 
software are called clerk nodes. 

Fig. 2 shows a clerk program 40 in a memory 44 of a node 41 . Clerk program 40 provides an interface 

30 between the network via lines 32 and an application program 42. which is also in memory 44 of node 41. 
Clerk program 40, which is executed by a processor 46 in the node 41 , transmits and receives information 
to and from network 29 via lines 32. Clerk program 40 communicates with application program 42 via 
locations (not shown) in memory 44 under control of application program 42. Many types of clerk programs 
similar to clerk program 40 may exist to handle the various needs of application programs. 

35 Fig. 2 also shows a local clock 48 of node 41. Local clock 48 is incremented regularly (If it is digital) or 
increased at a regular rate (if it is analog) to keep what is called "local time" for node 41. This incrementing 
is accomplished by an Increment circuit 49. Local clock 48 is accessed, i.e., set and read, by clerk program 
40 in node 41. Clerk program 40 also serves as an interface to local clock 48 for application program 42. 
Thus, clerk program 40 of Rg. 2 is called a "time service" clerk because it services requests for the local 

40 time from application program 42. 

As shown in Rg. 2. not all application programs interface to the network via clerks. Some application 
programs, such as an application program 50 of Fig. 2, handle their own network communications yet still 
access clerk program 40 to obtain the local time. 

Several methods exist for controlling the communication of information betveen the nodes 30. For 

45 example, the network of Fig. 1 uses a method called "carrier-sensing multiple access with collision 
detection" (CSMA/CD) as defined in IEEE Standard 802.3. Under CSMA/CD, all nodes on the network are of 
equal importance. A node 30 can "send" information on the network only when no other node is trying to 
send Information. If a node tries to send information at the same time that another node is also trying to 
send information, each node will wait a randomly-determined period of time and then attempt to resend its 

50 information. Since the period of time each node waits before attempting a resend is randomly detenmined, it 
is probable that both resends will be successful. Every node on the CSMA/CD network "listens" to all 
infonmation sent over the network. If any node "sees" information intended for that node, the node will 
receive the information and take appropriate action. Information passing between the nodes of the network 
is organized into packets. Each packet contains data or control infonmation sunrounded by control and 
routing information. 
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Network 29 of Fig. 1 operates using the Ethernet standard for data transmission. This standard governs 
the protocol used in transmitting data at a low conceptual level. It is compatible with the OS! and DNA 
communications protocols. 

Pacl<ets on an Ethernet network may be sent using one of two basic methods: unicastmg and 
multicasting. In unicasting, a packet is sent from a first node to a second node. The destination address 
field of the packet indicates that the packet is destined for a single node. In multicasting, a single node 
broadcasts information to multiple nodes and the destination address field of the packet indicates that the 
packet is destined for a group of nodes. 



b) REPRESENTATION OF TIME IN A DISTRIBUTED PROCESSING SYSTEM 



i) Notation 



/5 



20 



An abstract notion of time is defined by some standard which provides a frame of reference by which 
all values of time are related. One such standard is Universal Coordinated Time (UTC). an international 
standard maintained by the International Time Bureau (BIH) and the one in general use throughout the 
world. Political representations of UTC (such as Eastern Standard Time (EST) or Pacific Daylight Time 
(PDT)) are functionally equivalent to UTC. A clock is simply a device that provides a measure of UTC. The 
UTC time standard runs at a rate that is almost constant since it is based on ultra-stable atomic clocks. 
However, some users of time signals need time that is referenced to the rotation of the earth. This time 
standard is known as UT1 and is inferred from astronomical observations. To keep UTC and UT1 
approximately equal, occasional corrections of exactly one second - called "leap" seconds - are inserted 
25 into the UTC time scale when necessary. 

In order to quantify how well a clock is measuring UTC, it is important to define the concept of 
"correct" time. Let C, denote a local clock in an ith node of network 29 and let C(i.t) represent the value of 
Ci at the time t. (Throughout this specification, times denoted by t refer to UTC), In order to quantify how 
well a clock is measuring UTC, we introduce several properties of clocks. Most important of these is the 
property of con-ectness. The following definitions will be used throughout the remainder of the specification: 



30 



Inaccuracy: 

A perfect clock is one where CO,t) = t. vt. However, a real clock always introduces some error, 
inaccuracy of Ci at time t. denoted by a(l.t). is defined as the deviation of C? from UTC. Hence. 
a(i.t) = C(i,t) - 1. 



40 Drift 

The rate of change of inaccuracy is called the drift and for Ci of an analog clock is given by: 
5(i.t) = dC(i.tydt-1. 
Ci of a digital clock is given by: 
45 |C(i.t + At) - C(i.t) - At| < aAtaidt 
for some interval At. 



Monotonicity: 

50 . . w 

A clock is said to be monotonia if its measure of time is never decreasing, ue., it 
C(i,t2)SC(i,ti)fort2> ti. 



55 Resolution: 

The resolution of a . clock Is the maximum time interval that can elapse without there being any change 
in the value of the clock. The resolution is denoted by p. 
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Skew: 

For two docks i and j, the skew at time t is given by 
<y .i(t) = |C(i.t) - Ca,t)|. 

5 

Correctness: 

For connplete time information. It is not sufficient to supply only the time C(l,t). The inaccuracy must be 
10 represented as well. Suppose that a clock reports a positive inaccuracy l(i,t) with the value of time C(i,t). For 
this combination of time and inaccuracy to be correct we require that l(i,t) 2: (a(i,t)| meaning that 
C(i.t) • i(i.t) ^ t ^ C(i.t) + l(i.t). 

Time at clock i can be represented as an interval W(i,t) = [C(i,t) - l(i.t), C(i,t) + l(l.t)]. As shown in Rg. 3. for 
an arbitrary interval V = [x,y]. we define LvJ= x; ["vn = y. OVll = (x + y)/2 (the midpoint of the 
IS interval); and |V| y - x (the width of the interval). 

Using interval notation, the inaccuracy is given by l(i,t) =|W(i,t)|/2. A time interval is said to be correct if 
LW(i.t)J ^ t S Pwo.t)"!. i.e.. the interval contains UTC. A set of M local clocks is said to be synchronized if 

the clocks* inten/als overlap, i.e.. W(1.t)/) W(2.t) W(M.t) i* 0. If the clocks are conrect. they must be 

synchronized. 

20 Next, some notation for interval arithmetic is defined. The sum of two intervals is defined as (x,y] + 
[u.v] = [x + u, y + v]. The sum of an interval and a scalar is identical to the sum of two intervals, one of 
which has width zero. Thus [x. y] + z = fx.y] + [z.z] = [x + z. y + z]. This sum is equivalent to 
translating the interval along the real line. The notation [x. y] ^ z denotes a "stretching" of the Interval along 
the real line. The notation z = [x - z. y + z]. 

25 

ii) Clocks 

Two basic types of clocks exist: clocks that obtain time from an external source, such as radio signals. 

30 and clocks that are self-contained, such as crystal clocks. Clocks of the first type measure UTC with a 
known bound on the inaccuracy which depends on the propagation delay of the radio signal and 
implementation specific details. If operating correctly, the time and inaccuracy reported by clocks of the first 
type define an interval that contains the instantaneous value of UTC. Clocks of the second type supply the 
time in the form of a scalar and there is no inherent notion of inaccuracy. However, clocks of the second 

OS type do have bounds on their drifts, which is denoted by 5 max. So, given an initial inaccuracy 1| at time Tf, 
it is possible to compute the inaccuracy at the time that the clock reads T from 1 ^ l| + (T - T,) 5 max. Thus, 
a self-contained clock can represent time as an interval but this interval increases or "degrades" as time 
progresses. 

In previously existing systems, no provision was made for adjusting the time of a local clock except by 
40 resetting it with a new value. As a result, the time could jump forward or backward arbitrarily. Such quantum 
jumps in local time sometimes resulted in events having an ordering by UTC such as shown in Rg. 4a 
being timestamped to reflect a different ordering such as shown in Rg. 4b. 



45 c) USES OF TIME IN A DISTRIBUTED PROCESSING SYSTEM 

Time, in a data processing system, is used in at least three ways. A first use is to time and order 
events. In many processes performed by data processing systems (for example, event logging), it is 
desirable to be able to determine an order of occurrence for several events, A second use for time in a data 
50 processing system is to determine an interval of time that has elapsed between two events. Examples 
include system timeouts, Ethernet packet lifetimes, and I/O timeouts. A third use for time in a. data, 
processing system is to schedule future execution of events. If an event is to begin execution at a time near 
a scheduled time, an inaccuracy of the dock must be known and bounded within certain predetermined 
limits. 

55 Early data processing systems were centralized mainframe computers and usually maintained a time 
via a single system clock Integral to the system. Vartous operating system and application programs 
executed on the centralized system and accessed the system clock when they needed a current time value. 
An error in the time kept by the system clock was discovered by human Inspection and remedied simply by 
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manually resetting the system clock whenever the error in its time exceeded some predetermined error 
limit. For example, the local clock of a VAX data processing system, manufactured by D'S'^^' f ^'PJ^"* 
Corporation, may drift by as much as nine seconds a day. This could ^/ dn« °* T not usuaS^ 
seconds (54.75 hours) per year. Such errors, while inconvenient are easily detected and do not usually 

5 cause a major problem for a centralized data processing system. Although such errors cause the time kept 
bv the system clock to deviate from UTC. the time is still increasing and monotonic because the system 
contains only a single clock. Thus, events will still be ordered correctly and time ititervals and tinieouts 
while perhaps of slightly incorrect duration, will still be correct relative to other intervals and timeouts in the 
system Events may be timestamped incorrectly in relation to UTC. but their ordering will not be altered. 

to in contrast, disagreement between the local clocks in the nodes of a distributed processing system 
such as the network of Hg. 1. can cause catastrophic errors in the system. Assume, for example, that both 
node A and node B of Fig. 1 contain a local clock keeping a local time. A difference in the Umes kept by 
the clocks of the two nodes may cause a distributed software process to produce erroneous results 

In Figs. 5a-5f. node A and node B of a distributed processing system are to perform different steps of 

»5 the following process at UTC times timet and tlme2. respectively, 



UTC timel: 


copy file1 to file2 


(by node A) 


UTC time2: 


copy file3 to fi!e1 


(by node B) 



where UTC timel precedes UTC time2. In this algorithm, the order of performance of the steps is cntical. 

If as shown in Rg. 5a. the clocks of both nodes A and B are keeping local times that are the same as 
UTC the copy from filet to file2 is performed first (by node A at node A's local timet) and the copy from 
filea'to filet is performed second (by node B at node B's local time2). This is the desired order of 

25 performance, yielding the results labeled "Results after Step 2" m Rg. 5b. 

If however, the local clock of node B is keeping incorrect time, erroneous results may occur If as 
shown in the timing diagram of Rg. 5c. the local clock of node B is being incremented 
the local clock of node A. then the local clock of node B may reach a local time2 before the <ocal clock of 
node A reaches a local timet. Similarly, as shown in the timing diagram of Hg. 5d. the local dock of node 

30 B may have been mis-set at some previous time. Therefore, even though the local clock of node B is being 
incremented at correct inten/als. local time2 of node B precedes local timet of node A. Alternately, as 
Shown in the timing diagram of Fig. 5e. the amount by which the focal clock of node B is ^^-"9 '"jemented 
may be larger than the amount by which the amount by which node A is being incremented. Again, such a 
condition will cause local time2 of node B to precede local timet of node A. ^^^^rr^c^^ fi«t tov 

as If any of the conditions of Rgs. 5c. 5d. or 5e exist, the copy from file3 to file! is Performed first Jy 
node B) and the copy from file1 to file2 is perfomied second (by node A). This inversion of the order of the 
Steps performed by nodes A and B causes the erroneous results labeled "Results after Step 2 in Rg. 5f. 

What is needed is a method for maintaining a correct time in a data processing system where the 
method ensures that the local clocks of the nodes in the system agree on a correct time. j^reement among 

^ the nodes on some random time is not acceptable since the agreed upon ^-^f J^^V J=''<2'/'°"'.,Tlf.«t 
system where all the nodes agree on a non-UTC time will avoid the problems of Rgs- 5a-5f. but will cause 
errors in time-stamping functions, and will not execute scheduled events at the expected times. In additon 
to maintaining a correct time, such a system should be able to survive the loss or errorieous operator, of 
one or more nodes of the system and to identify and correct these faulty nodes if possible. The times kept 

« by the local clocks of such a system shouW always be monotonic and Increasing to avoid the problems of 
Rgs. 4a-4b. 
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SUMMARY OF THE INVENTION 



The present Invention overcomes the problems and disadvantages of the prior art by providing correct 
time from server nodes to clerk nodes at the request of H^e dsrk nodes. ^ . " 

In accordance with the purpose of the invention, as embodied and broadly descnbed herein, the 
invention comprises a method for maintaining correct time in a cierk node in a distnbuted processing 
ss system also containing a plurality of server nodes. The clerk and server nodes are connected together and 
the sewer nodes provide time infomation to the system. Each of the nodes includes a local clock keeping a 
local time and each of the local clocks has a local drift value assodated therewith representing a bound on 
the rate at which that local time deviates from a correct time value. The method compnses the steps 
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performed by the clerk node of: requestin g an updated time interval f rom at least one of the server nodes 
when the local Inaccuracy of the local clock of the clerk node ^ceeds a predetermined maximu m 
inaccuracy value: noting a request time according to the local clock of the clerk node when the clerk node 
requests the updated time Interval; receiving from the server nodes respective updated time interval 

5 representations and corresponding delay values: noting, for each receipt of the updated time interval 
representations and corresponding delay values, a different response time according to the local clock of 
the clerk node; calculating a correct time interval from the received updated time interval representations, 
the received delay values, the noted request time, the noted response times, and the resolution value of the 
local clock of the clerk node, wherein the correct time Is contained within said calculated correct time 

10 interval: and adjusting the locaj time kept by the local clock of the clerk node according to the calculated 
correct time interval. 

The invention further comprises a method for maintaining a correct time in a clerk node including the 
steps performed by the server node of: receiving a request for an updated time interval from another node 
when the local inaccuracy of the local clock of the other node exceeds a predetermined maximum 

15 inaccuracy value; noting a receipt time according to the local clock of the server node when the server node 
receives the request from the other node; calculating a delay value from the noted receipt time; calculating 
an updated time interval representation from the local time of the local clock of the server node and the 
inaccuracy of the local clock of the server node: and sending the updated time interval representation and 
the corresponding delay value to the other node. 

20 The accompanying drawings, which are incorporated in and constitute a part of the specification, 
illustrate one embodiment of the invention and. together with the description, serve to explain the principles 
of the invention. 

25 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a general block diagram of a local area network. 

Rg. 2 is a block diagram of a clerk node which can be connected to a network. 
30 Fig. 3 is a diagram used to explain terms which define a time interval. 

Figs. 4a and 4b are diagrams used to illustrate a problem which may be caused by certain methods 
of resetting clocks in nodes of a network. 

Figs. 5a-5f are diagrams used to Illustrate a problem which may be caused by timing malfunctions In 
nodes of a network. 

35 Ftg. 6 is a block diagram of a time maintainer node of a network in accordance with this invention. 

Rg. 7 is a block diagram of a time provider node of a network in accordance with this invention. 
Fig. 8 is a flow chart of one method in accordance with this invention of adjusting a local clock in a 
clerk node of a network. 

Rg. 9 is a flow chart of further steps of the method of Fig. 8 In accordance with this invention of 
40 adjusting a local clock In a clerk node of a network. 

Rg. 10 is a diagram illustrating the time delays In a request from a clerk and a response from a 

server. 

Rg. 11 is a flow chart of the preferred procedure by a clerk for requesting a time update from a 
server. 

45 Rg. 12 is a flow chart of part of a preferred method of the present Invention to compute the smallest 

interval guaranteed to contain UTC. 

Rg. 13 Is a flow chart of a preferred method of efficiently Finding the minimum endpoint of the best 
interval. 

Rg. 14 is a flow chart of a preferred method of the present invention for adjusting for leap seconds. 
so Rg. 15 is a flow chart of a preferred method of the present Invention of how a server node responds 

to a request for time from a cierk node. 

Rg. 16 Is a flow chart of a preferred method of the present invention for manually resetting time 
maintainer nodes. 

Rg. 17 Is a flow chart of a preferred method of the present invention for detecting faulty server 

55 nodes. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 
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Reference will now be made in detail to preferred embodiments of this invention, examples of which are 
illustrated in the accompanying drawings. Whenever possible, the same reference numbers will be used 
throughout the drawings to refer to the same or like parts. 

5 

a) HARDWARE OVERVIEW 

The method of the present invention is preferably earned out by data processing hardware executing a 
plurality of software programs. That hardware is preferably contained in nodes of a network such as network 
,0 29 Shown in Fig. 1. As shown in i^g. 2. a node 41 which executes clerk program 40 .s called a clerk node. 
The purpose of clerk node 41 Is to service requests for a local time of local clock 48. As explained above, 
information passes from the connecting lines 32 into the clerk node 41 and to clerk program 40 executing 
on clerk node 41. Clerk program 40 updates its local clock 48 based on infomnation received o^er lines 32 
and. in response to requests from application program 42. reads the local time from local clock 48 and 

IS supplies it to application program 42. » , ^ ^, An anH 

Preferably, other nodes of the network execute similar software programs to clerk program 40. and 
supply time information to other application programs. 

In accordance with ttie present invention, other nodes, called server nodes, execute software programs 
called server programs. Sen^r programs send time information to clerk and server nodes over lines 32^ 
20 Preferably, there are two types of server nodes: time maintainer nodes (also called time maintainors) and 
time provider nodes (also called time providers). ^ „.^ . *^ 

Fig. 8 shows an embodiment of a time maintainer node 80. Time maintainer node 80 is connected to 
the network via lines 32. Similar to clerk node 40. time maintainer node 80 contains a processor 82 and a 
local clock 84. Local clock 84 contains an incrementation circuit 88 for penodically increasing the local time 
25 of local clock 84 by a predetermined resolution value. 

5pon receiving a request for an updated time from a clerk node over ttie network, processor 82 
executes a server program 86, which is stored in a memory 87 (preferably RAM) of time maintainer node 
80. Server program 86 sends time information from server node 80's local clock 84 a to requesting clerk 

node over the network. . , ^ 

30 Some time maintainer nodes may also Include a battery back-up clock and non-volatile memory. The 
non-volatile memory stores a base local time and inaccuracy at a time T,. which can be recovered in case 
of system failure. Similarly, the battery back-up clock continues keeping a local time during system failure^ 

Fig 7 shows a second type of sender node 80 called a time provider node. Like time maintaner node 
80 of Rg. 6. time provider node 80' contains a processor 82 and a local clock 84 . A sen/er program 86 . 
which is stored in a memory 87' of tiie time provider node 80 and executed by processor 82 of the node 
80'. sends time infonnation from ttie local clock 84 over the network upon receiving a request from a clerk 

"° Preferably a requesting clerk node does not perceive any difference between time maintainer node BO 
and time provider node 80'. Clerk programs use identical protocols to communicate with bo«i types of 
40 nodes and ttie information received by a clerk program from botti types of senrer nodes preferably has the 

^"^ThelTey' difference between a time maintainer node, such as node 80. and a time provider node, such 
as node 8o'. lies in the way those nodes update their local clocks. Time maintainer node 80 updates its 
local clock using tiie same metiiod as clerk nodes, which is described in detail below. Time provider 80 

45 updates its local Clock 84' by accessing an external time source. u- u • » 

As shown in Rg. 7, time provider node 80 contains a radio ti-ansmitter/receiver 94 which is used to 
receive time signals from a UTC broadcasting source, such as the WWVm Denver. Colorado. These 
received time signals are used by sen/er program 86 to update local clock 84 of time provider node 80 . It 
should be noted that time provider nodes could alternatively access some other independent time source 

50 outside the network that provided time information not contaminated by any system delays, dnfts of the 
local clock, or otiier inaccuracies. Preferably, a time provider node that cannot access its external source 
will alternately update its local clock using the same method as time maintainer and clerk nodes. 



96 



55 b) OPERATION OF CLERK NODES 

A method according to this invention will be described with respect to clerk node 40 shown in Fig. 2. 
although it shouW be understood that a network using the present invention may include other clerk nodes 
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operating according to the method described below. The description of the operation of clerk nodes in 
accordance with the present invention will be presented in two parts. The first part describes how a clerk 
maintains its local clock and knowledge of the inaccuracy of its clock. This first part includes the method by 
which a local clock of a clerk node is correctly and monotonically adjusted to compensate for drift. The 
5 second part describes how a clerk node resynchronizes (obtains a new estimate of the time from the server 
nodes). This second part describes under what circumstances a clerk resynchronizes. the protocol for 
obtaining time estimates from server nodes, and an algorithm for computing an optimal correct time interval 
from these estimates. 

ro 

') Monotonicaily Adjusting the Local Clock in Clerk Nodes 

A local clock, such as local clock 48 of Fig. 2, preferably comprises a software register representing the 
current time. This register preferably is updated by incrementation circuit 49. which may comprise a service 

75 routine invoked by a hardware interrupt of a periodic timer. In such an embodiment, these interrupts are 
called "ticks." The time between ticks determines the resolution p of the local clock, which, in a preferred 
embodiment, equals 10 ms. An interrupt sen/ice routine services tick interrupts and increments the local 
clock by p. By changing the amount of the increment while keeping the time between ticks constant, the 
clock drift can be artificially altered, causing local clock 48 to gain or lose time. Alternatively, one could alter 

20 the time between ticks while keeping the software increment at a predetermined value. 

The preferred manner by which clerk node 40 adjusts its local clock 48 to a new time can be described 
by the procedures of Rgs. 8 and 9. Rg. 8 shows a flow chart for a procedure called "AdiustClk_Start," 
which is executed at a "resynchronization time." Tr, to begin incrementing the local clock by a new 
incrementation value instead of by the predetermined value p. Rg. 9 shows a flow chart for a procedure 

25 called "AdjustClk_End," which is executed at a time T^ + D (where D is the duration for which the new 
incrementation value is used) to end incrementing the local clock by the new incrementation value and 
resume incrementing by p. 

In Rgs. 8 and 9, clerk program 40 has determined that the actual time will be <Ta,la> when local clock 
48 reads Tr. "Maxdrift" represents the predetermined minimum number of seconds during which the local 

30 clock will normally drift one second. The notation <Ta,la> represents a time interval [Ta - Ta + U] having 
time ta as a midpoint and extending an inaccuracy distance la in both a positive and negative direction. 
Thus, maxdrift = t5max. The value "k" represents a predetermined drift increment rate, i.e.. the rate at 
which the drift of the local clock will be artificially increased or decreased during the interval [Tr.Tr + D]. In 
the preferred embodiment, k ^ -1 . In other words, a local clock cannot be decreased at a rate of more than 

35 one second per second or it will run backwards. \n Rgs. 8 and 9, "adj" represents a predetermined number 
of seconds during which the drift of the clock is altered by one second. (Thus, adj = 1/|k| and. in the 
preferred embodiment, adj ^1.) "Irate" represents a rate at which the inaccuracy of the local clock increases 
in response to the drift of the local clock. Under normal operation. Irate = Srnax, i.e., Ute equals the 
maximum drift of the local clock. However, while the local clock is being adjusted Irate is decreased to 

40 compensate for the drift increment rate, k. 

Rrst. as shown in Rg. 8. at the resynchronization time T,, a current time, T|, Is set to T^ (Step 100). An 
inaccuracy, Ij, corresponding to Ti, is set to: 
li = fa + jTa-Tl| 

(Step 100). where la is the inaccuracy of the new time value and |Ta - TJ is an additional inaccuracy 
45 representing the difference between the local clock and the actual time. 

!f Tr is equal to Ta (Step 104), then the local clock is already set correctly and no further adjustment is 

needed. If Tr is not equal to Ta. however, and if Ta is greater than T, (Step 106), then k, the rate at which to 

alter the drift of the local clock Is positive (Step 108) so that the local clock will gain time. Otherwise, k is 

negative (Step 110) so that the local clock will lose time, 
50 Next, irate, the rate at which the Inaccuracy of the local clock increases in response to the drift of the 

local clock, must be set to reflect the changed drift value of the local clock. Irate is calculated according to 

the following formula: 



55 ^rate * adi - maxdrift 

adj * maxdrift 
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(Step 112) Because, in the preferred embodiment, the absolute value of the adjustment rate, |k|. is greater 
than the drift rate, 5max. adj < maxdrift. i.e.. 1/k < l/5max. and therefore l,a,e is negative and li. the 
inaccuracy before adjustment, is less than h', the inaccuracy after adjustment. 
"D." the duration of the adjustment is computed in Step 114 using the formula: 

^ LcauUVd^equals'the predetermined number of seconds during which the drift of the local clock is altered 
by one second and because IT3 - T^l equals the total number of seconds by which the clock is to be altered 
the product of adj and |Ta - TJ yields D. The procedure AdjustClk_End is then scheduled to execute at 
time Tr + D (Step 116) and clock increment circuit 49 begins incrementing local clock 48 by (1 + k)p 

TO (Step JJ®^^^^^^^^^ ^^^^ procedure AdjustClk_Start completes execution before the next tick after T,. 

Otherwise, the procedure would have to compensate for its own execution time. 

For the purpose of example, suppose that the variables p, k, 5max. adj. maxdrift. Ta. U. and Tr have the 

following values: 
;5 p = 1-2 sec 

k = 4/5 sec per second 

5max = 1.4 sec per second 

adj = 1/k = 5/4 sec 

maxdrift = 1 amax = 4 sec 
20 Ta = 5 

la = 3 
Tr = 1 

Then. Ii = U + |Ta • T,| 
= 3 + |5 - 1 1 =^ 7 sec 



Irate ' ad1 - maxdrift 
adj * maxdrift 



30 



5/4 - 4 « -11/20 sec per second 
5/4 * 4 



D = |(T. - Ti)| " adj 

In Stti^'wo^dsffor tocal clock 48 to gain 4 seconds, its drift must be increased by 4/5 seconds for a total 
duration of 5 seconds. Thus after 5 seconds (n = 10 increments). 
Ti' = T, + n(1 + k)p 

= 1 + 10(1 + 4/5)1/2 = 10. . u ^ ^ 

Local clock 48 has gained 5 seconds due to normal incrementation and 4 seconds due to its altered drift 

rate. — 
^ The Steps of Fig. 9 return local clock 48 to its regular incrementation procedure. At time T, + D, a new 

inaccuracy, \\ is computed according to the formula: 

{Step'*l22)^whelre hT^the inaccuracy from Step 100 of Fig. 8, T* is the cun-ent value of local clock 48. T, is 
the time from Step 100 of Fig. 8. and Ute is the increase in the inaccuracy from Step 112 of Fig. 8. In the 
^ current embodiment, because Vaie is negative. U < l|. Both T| and I, are stored in memory 44 to be used 
by clerk 40 In determining a time to resynchronize as described below. 

Therefore. In the above example, the new inaccuracy of local clock 48 is: 

i|' = h + (Tl' - Ti) ' Irate 

= 7 + (10 - 1) • -.55 = 7 - 4.95 = 2.05. 

In Step 128. the amount to alter the clock's drift, k, is set to zero because the clock adjusting procedure 
is done. Similarly, the rate of increase of the inaccuracy is reset to its original value of 6max in Step 130. 

The procedures illustrated in Figs. 8 and 9 thus guarantee that local clock 48 remains monotomc. 
However, this may not always be desired behavior. In particular, if local clock 48 differs from the corre^ 
time by a large value, it may be preferable to simply reset it to the correct time rather than to adjust its dnft 
using the procedures of Figs. 8 and 9. 

ii) Determining When Clerk Nodes Should Obtain Time Estimates 

9 
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Clerk program 40 resynchronizes its local clock 48 when its inaccuracy exceeds a predetermined value 
Imax. If a clerk program has resynchronized to a correct time Tg with inaccuracy the inaccuracy at any 
later time T is given by I = Ig + (T -Ta)lrate. where Irate = Bmax. However, the preferred embodiment does 
not guarantee any bound on Uax- As a result, it is possible that Imax - U is small or even negative. To 

5 prevent clerk program 40 from synchronizing continuously, the preferred embodiment computes the next 
time to resynchronize, T, according to the following rule (where "max" is a function that selects a maximum 
of two values and "Synchold" is a predetermined value representing the shortest time allowed between 
resynchronizations in the preferred embodiment). 
T = Ta + max(synchold, (Uax - laV^max. 

10 (Imax - laK^nnax represents a time when the inaccuracy of local clock 48 will equal or exceed Imax- 



ill) Obtaining Time Estimates from Server Nodes by Clerk Nodes 

rs In the preferred embodiment of this invention, clerk program 40 obtains the time from a particular 
server node by making an explicit request to the server program of that node for the time and then waiting 
for a response. This method of access has two features. Rrst. it makes it possible for a clerk to measure 
the delay incurred due to the communication protocol overhead and, hence, makes it possible to bound the 
inaccuracy of the received time. Second, it requires no specific underlying communications hardware. 

20 Fig. 10 is a time-space diagram of a request from a clerk, such as clerk program 40. and a response 
from a server program, such as server program 86 of time maintainer node 80 or server program 86 of 
time provider node 80'. The times ti, I = 1 ,2...a. con-espond to the values of UTC (which can never be 
known exactly due to the delays discussed above.) The value of the clerk's clock at time ti is denoted by 
Cc(ti) and the value of the server's clock at time ti is denoted by Cs(ti). A double arrow denotes a message 

25 transmission and a single arrow denotes a message reception. 

Fig. 11 is a flow chart of a preferred method performed by the clerk node for making the request 
illustrated by Rg. 10. Once the clerk decides that It is time to resynchronize (Step 200), at ti, the clerk 
reads its local clock which returns the value Cc(ti) (Step 202). The clerk then initiates a request to the 
server (Step 204) the request is transmitted at time t2 after an unknown sending delay Sc, as shown in Fig. 

30 10. and the request is received by the server at time t* after unknown delays of Xcs and rg. 

The clerk waits until it receives a response from the server (Step 208) in the form of an interval [u,v], 
and a delay w. (In a preferred embodiment, the inten/al [u.v] is represented by a midpoint value and an 
inaccuracy value. Other embodiments could represent [u,v] by two endpoints.) The response from the 
server is received at time Cc(t8) (Step 210) after incurring random delays similar to those of the request, 

35 namely Ss. 7" sc. and Tc- 

Ideally, the clerk would compute the time Cs(ti) from the value Cs(t4) returned with the response as 
follows: 

Cs(ti) = Cs(U)-x, 

where x = Sc + Tcs + ^s- However, x is the part of the delay that cannot be known exactly. It is in the 
40 range 0 S x S ts - ti - w. If x « 0. then Cs(ti) = Cs(U) while If x = ts - ti -w. then Cs(tt) = Cs(U) + w - (Cc- 
(ta) - Cc(ti)). Hence, at best, the time seen by the clerk at ti can be bounded by 
Cs(t4) + w - (Cc(t8) - Cc(ti)) ^ Cs(t,) ^ Cs(U). 

but it can never be known exactly. The effect of the drift of the clerk with respect to the server over the time 
period of the exchange is neglected. Clearly, by minimizing the components of delay, or by compensating 

45 for known delays such as w, the interval bounding Cs(ti) can be reduced. 

Since local clock 48 is discrete, the timestamp Cc(t8) actually corresponds to the value of local clock 48 
in the range [Cc(t8),Cc(t8) + p], where p is the resolution of local clock 48. Furthermore, the time provided 
by server i is In fact an interval W{\M) = [u.v] rather than a scalar value. Thus, the time interval of Si at 
time ti as seen by the clerk is given by: 

50 U(i.ti) = [u + w - (Cc(t8) + P - Cc(ti)).v] 

In the preferred, embodiment of the invention, the clerk complies U(l,ti) in this manner (Step 212). 

The value w, which is returned with the time to the clerk, conresponds to the known delay at the server 
after the time which is reported to the clerk (and hence, It may include any deterministic component of Ss). 
The server does not include any deterministic component of rs in the value of w returned as this delay 

55 occurs before t4.. Furthermore, there is no need to return the value of known delays before t^ since the 
server can include this information by decrementing the time returned O-®-* Cs(t4)) and by Incrementing w 
by the same amount. It would not be incorrect to report a value of w less than the actual delay incun-ed. 
This smaller value of w would merely cause a larger than necessary inaccuracy in local clock 48. It would, 
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however, be incorrect to report a value of w greater than the actual delay. Th.s larger value of w would 
cause the inaccuracy of local clock 48 to be smaller than could be justified by the actual delay '"curred by 
the server. To maximize performance, w should reflect the delay at the server as accurately as possible, but 
w should never be greater than the actual delay incurred. 

Since request and response packets may be lost or servers may become unavailable, a clerk waits only 
a predetermined time for a response to a request 



Iv) Computing the Correct Time from Received Time Estlrnatgs In Cjerk Nodes 

Flo 12 is a flow chart of the preferred method used by clerk program 40 in accordance with of the 
present invention to compute the smallest interval guaranteed to contain UTC. Assume that clerk C, obtains 
the time from M servers by using the method in Fig. 11 and therefore has a !nd Uf^T^ i.'the 

1.2...M. Tj is the value of the clerk's clock at some arbitrary instant t,- fi.e.. T, - C(i.tj)> and UO.Tj) is me 
js value of the j*" server's clock at tj. as seen by the clerk. , . ^ « »♦ 

First, procedure AdjustCIk-End of Fig. 9 is invoked (Step 250). This ensures that the clock is drifting 
a rate less than maxdrift (i.e.. k = 0). Next, for each interval received from a server, its value at the next 
resynchronlzation time T, is computed from 
Ua.Tr) = UO.Tj) + (T, - Tj) i (Tr - Tj)«max 

20 (Step 252)^^^ endpoints of the intervals U(j.T,) are arranged into a list in ascending order (Step 254). The 
list is of length 2M because there are two endpoints for each interval in the list. If two or rnore endpom s 
have the same value, those intervals corresponding to lower bounds must proceed those intervals 
correspond'g uppe; bounds In the list. That is. if LU(i.T,)J = ruO-T,)"! then LUO.TjJ must preceed 

nI^ rcountel; f. representing the number of existing faulty servers is set to zero (Step 256) and the 
list IS scanned in ascending order to find the first endpoint that is contained in at least M - ["^'^^^^l^^^P 
258) This point, if found, is a lower endpoint of the smallest time interval containing UTC. If no such point is 
fourld then there are more than f faulty servers. The counter f is incremented (Step 2eO). control retums to 
30 Step 258. and a smaller number of intervals Is scanned for a common endpoint "/f J^.^' ®"JP°'"; .'^ 
found (Step 258). then the list is scanned in descending order to find a second endpoint that is contained in 
at least M - f intervals (Step 262). This point is an upper endpoint of the smallest time interval containing 

*^^%anning the list for the first upper and lower endpoints is accomplished more easily if tiie elements of 
the list are marked as maximum end points or minimum end points depending on whether they conrespond 

*° ^Fii'^VcZh^ilsai^^^^ a method for finding the lower endpoint of the best correct time 

ir^terval efficiently. This method is used in Step 262 of Fig. 12. Rrst. two counters, '"i^ J«f "^tX« 
Mostlnts. are initialized to -I." Intersecting Ints represents a number of intervals containing a tenta*y« 
minimum endpoint. Mostlnts represents number of intervals containing a current minimum endpoint A third 
loop counter, i. is initialized to -2" to indicate that the second list element is being considered as the 
tentative minimum endpoint The first element in the ordered list of endpoints of Step 254 ^f Fig. 12 is 
designated as the current minimum endpoint by setting Minpoint to "1 " (Step 270). Thus, the initial value of 
IntersecUnglnts indicates that only one interval thus far contains the tentative minimum endpoint indicated 
by the counter i. The initial value of Mostlnts Indicates that the first endpoint in the is^ (designated by 
Minpoint) is contained in one internal. Next, for increasing values of i through 2M.1. the following Steps are 

^^f^r'tth list element is greater than the tentative minimum endpoint designated by Minpoint (Step 272). 
then the counter Intersectinglnts is incremented (Step 274). and a test is performed to determine if the 
counter Intersectinglnts is greater than the counter Mostlnts (Step 276). If yes. the c«5unter Mostlnts is set to 
Intersectinglnts. the number of intervals containing the ith list element is saved, and the current value of i is 
saved as the tentative minimum endpoint (Step 278). Otherwise, if Intersectinglnts is not greater than 

Mostlnts. control passes to Step 282. „„..„tor 
If the ith list element is not greater than the tentative minimum endpoint (Step 272). then the counter 
Intersectinglnts is decremented (Step 280) and control passes to Step 2ffi. ^ u„ « 

At Step 282. a test is performed to detemiine if more endpoints in the list have yet to be examined. If 
yes. i is incremented (Step 284) and Steps 272 through 282 are repeated to examine another endpointH 
no. then ail of the 2M.1 elements In the list of endpoints have been checked and control passes to Step 
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286. At Step 286. a test is performed to determine If an endpoint within M-f intervals has been found. If yes, 
a minimum endpoint has been found (Step 288). Otherwise, no endpoint is contained in at least M-f 
intervals (Step 290). A similar algorithm can be used to find the maximum endpoint of the best con-ect time, 
as required by Step 262 of Fig. 1 2. by searching the endpoint list In descending order. 

The result of the method of Figs. 12 and 13 is the time (with inaccuracy) that the clerk's clock should 
read when the clock value becomes Tr. i.e., <Ta.la>. A clerk preferably uses the times from all the servers 
that are available. Other embodiments, however, may use the times from fewer than ail of the servers. 



V) Leap Seconds In Clerk Nodes 

All clerks become faulty immediately when leap seconds occur unless their Inaccuracies are sufficiently 
large so that their Intervals contain UTC both before and after the leap second. To ensure correctness in the 
presence of leap seconds, clerks must increase their inaccuracies by one second whenever a leap second 
might occur. Unfortunately, leap seconds may occur at any point in time, the only requirement being that at 
least eight weeks warning be given by the International Time Bureau (BIH) of an impending leap second. 
However, when choosing the time of a leap second, BIH is required to give preference to the last second in 
the months of June and December. Secondary preference is given to the last days of March and 
September. To simplify the matter, the preferred embodiment assumes that leap seconds occur at all four 
possible times. 

Fig. 14 Is a flow chart for making adjustments for leap seconds by clerk program 40. If a current time, 
T, of local clock 48 plus a current inaccuracy, I. of local clock 48 Is midnight of either March 31. or June 30. 
September 30, or December 31 (Step 300) then the current inaccuracy, I, is Incremented by "1" (Step 304). 
This adjustment is always performed whether or not a leap second actually occurs. Other embodiments 
may compensate for leap seconds only In response to a command from a human operator. 



c) OPERATION OF SERVER NODES 

A description of the operation of server nodes consists of four parts. These parts are: how servers 
initialize and operate; how time maintainor nodes periodically obtain a new time from other servers; how 
time maintainer nodes can be manually reset; and how servers periodically check for faulty servers. 

^ssponse to Clerk Node's Requests by Server Node 

Fig. 15 is a flow chart performed by a server In responding to a clerk node's requests, as demonstrated 
in Fig. 10. As shown in Fig. 15. a server waits until it receives a request from a clerk (Step 220) and notes a 
time of receipt C6(t4.) (Step 222). This time of receipt is later than the actual arrival time of the request by r,. 
The server processes the request by computing its own inaccuracy at time Cs(t4) and by noting a known 
component of delay w (Step 226). Lastly, the server transmits the local time interval and w to the requesting 
clerk at time Cs(t6) (Step 228). This action takes a time Ss. As explained above, delays rs and Ss are random 
and unpredictable. 



11) Initialization of Server Nodes 

A time provider initializes by obtaining the time from an external source. It then attempts to discover 
other sen/ers, by means of the protocols described below, for the purpose of periodically checking for faulty 
servers. 

When a time maintainer initializes, it first attempts to discover other servers. -aa^descrlbed below. If 
other sen/ers are available, it obtains the time from them as if it were a clerk node. If no other servers are 
available (meaning that the initializing server is the first server to be started), but the time maintainer has a 
battery back-up clock and non-volatile memory in which T| and I| are stored (as shown In Rg. 6), the time 
maintainer sets its local clock to the time of the battery clock and restores T| and 1} from the stored values. 

If the timer maintainor does not have a battery clock, it waits either for other server nodes (such as time 
providers) to become available or for a command from a human operator to reset the time. In the latter 
case, the time mainteiner simply accepts the new time without recourse to any consistency checks. 

12 
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iii) The Synchronization Algorithm for Server Nodes 

Time maintainers resynchronize periodically to compensate for the drift of their local clocks and to 
reduce the width of their time intervals. A time maintainer determines when to synchronize by the same 

5 method used by clerks, which is described in connection with Rgs. 10-13. When resynchron.zing. a time 
maintainer also performs the algorithm described betow for detecting faulty servers and. in addition adjusts 
ITs dock to the correct time in the same way that clerks do (see Figs. 8-9). In contrast, time providers do 
not resynchronize to other servers because time providers obtain time from an external source. The 
inaccuracy of the time obtained by time providers is implementation specific because it depends on the 

10 hardware used to obtain the external time. 
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)Manual Resetting of Time Maintainer Nodes 
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,5 If there are no time providers on the network, there is nothing to limit the extent to which the 
inaccuracies of the time maintainers increase. Thus, after long periods of time (for example, on the order of 
weeks) the inaccuracy of a time in a network may be large (on the order of minutes). To prevent the 
inaccuracy from becoming undesirably large, a manual reset method exists in the preferred embodiment 
whereby the time can be updated by resetting the time at single time maintainer To protect ^9^"^^ ^ 
20 human operator erroneously supplying an incorrect value of time, the manual reset method 3. es he 
time interval supplied by the human operator to be contained in the local time inten/al of the server being 
reset. Thus, the manual reset method described below does not allow large changes to the current time 
kept by a local clock. It does, however, allow a human operator to reduce the inaccuracy of a local clock or 

to make small changes to the value of the clock. , , 

Fio 16 shows a flow chart of the algorithm that a preferred embodiment of a time maintainer node 
executes in response to the receipt of manual reset request including a new time inten^al. The new ftrne 
interval is denoted by <T™. I„> and the time at which the manual reset request is received, as measured by 

the local clock, is denoted by Tu. 

First, the correct time is computed as described in connection with Figs. 12 and 13 (Step 320). This 
Step determines the actual time <T,. !,> corresponding to the future instant at which the local clock will 
read T. The interval <T„. i^. which con-esponds to local time T„. is translated to ite - 
l>. corresponding to local time T,. according to the formula given above in connection with Step 252 of 

"^'^'if <t!!''|> not contained in the interval <T,. \^ (Step 328). the manual reset request is rejected 
35 and an error Is noted (Step 330). Otherwise the request is accepted and control passes to Step 3^ where 

TX seTtoT'l and I, is set to l'^. When the local clock reads T,. procedure AdjustClk-Start shown ,n F.g. 8 

is'initiated to adjust the local clock of the time maintainer node (Step 334). . ic v^tr^rc 

The time maintainer must complete the adjustment accomplished by the Steps shown in Hg. 16 before 

resynchronizing to other servers. Otherwise, the manual reset request may not take effect. It is only 
40 necessary to reset a single time maintainer. All other time maintainers will conform to the new time once 

^''^^Whln^rese^^^^ the time, a human operator is expected to supply a correct inaccuracy. If no Inaccuracy 
is supplied, a predetermined default value is used. Human operators resetting the time must supply a 
correct inaccuracy to within a predetermined inaccuracy to ensure correct operation of the system. 

v) Epochs and Epoch Numt>ers in Server Nodes 

Occasionally, the time kept by the nodes in a network is Incorrect due to operator en-or or some 
catastrophic failure and not to failures in the servers themselves. When all nodes are keeping an incorrect 
time It is impossible to reset the time using the manual reset request described above. Instead, it is 
nece'ssary to shut down all the servers and restart them with a new. correct time interval. This process, 
however, may not be possible if a faulty remote server falls to shut down. To recover from this conditoon 
the present invention employs epoch numbers which allow groups of servers to be <1«»?P'®J'.^*^ J^^'";^ 
55 make it possible to introduce a new value of time into the distributed system by introduang it into one of 

the each server is assigned an epoch number. Servers preferably include 

their epoch numbers in their responses to requests for the time. Clerks always ignore this parameter. 
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However, when a server resynchronlzes or checks for faulty servers, it ignores servers with epoch numbers 
different from its own, as shown in Step 402 of Fig. 17, thereby decoupling that server from servers of a 
different epoch. A server epoch number can only be changed by a human operator. Whenever a new epoch 
number is assigned to a given server, that server must reinitialize according to the procedure described 

5 above, A human operator changes the epoch number at a single server and supplies it with the correct time 
If the server is a time maintainer node. Then, the human operator can migrate the other servers to the new 
(correct) epoch by changing their epoch numbers, one at a time, to correspond to that of the new epoch. As 
each server's epoch number is changed, it will reinitialize and obtain the correct time. To assist with the 
management of epoch numbers, each server notes any servers whose epoch numbers are different from its 

70 own. 



vi) Detecting Faulty Server Nodes 

75 Preferably, server nodes periodically check to see if other server nodes are faulty but operational. The 
method used is to obtain the time from all available servers, compute the best correct time interval, and see 
if any senders do not agree with the correct time. The details of the prefenred algorithm shown in the flow 
chart of Fig. 17. 

First, the time is obtained from all available servers including the time from the clock of the server node 
20 executing the method (Step 400). Time intervals obtained from servers of a different epoch (Step 402) than 
the executing server node (see below) are discarded and logged (Step 404). 

The conrect time interval corresponding to the resynchronization time T^ is computed using the 
algorithm presented in connection with the resynchronization of clerk nodes (Rgs. 12-13). and the value of f, 
representing the number of faulty servers resulting In this interval, is recorded (Step 406). If the value of f is 
25 not zero, there are faulty servers (Step 408) and an event including the identification of all servers and their 
respective times is logged (Step 410). If the executing server's own time interval is not contained in the 
computed correct interval, then the executing server is faulty (Step 41 2). In this case, the executing server 
must refrain from responding to requests for the time until it has corrected itself (Step 414). A faulty time 
maintainer corrects itself by adjusting its clock as described above (for clerk nodes) in Figs. 8 and 9. A 
30 faulty timer provider attempts to conrect itself by obtaining the correct time from the external reference. 

Time maintainers preferably execute the procedure of Rg, 17 whenever they resynchronize. Time 
providers preferably execute the procedure at predetermined intervals. 



35 vii) Leap Seconds in Server Nodes 

The operation of a time maintainer node to accommodate leap seconds is identical to that of clerk 
nodes as described above with respect to Fig. 14. In some embodiments, time providers may include a 
local clock which is kept correct by periodically polling a device that receives the external signal (such as a 

40 WVs/v receiver). For the purpose of accomodating leap seconds, such a time provider node must make 
adjustments for leap seconds using the method used by time maintainers and clerks. More specifically, at 
the time of a potential leap second as measured by T + I, the inaccuracy must be increased by one 
second by a method similar to that of Rg. 14. 

Other embodiments of the Invention will be apparent to those skilled in the art from consideration of the 

45 specification and practice of the Invention disclosed herein. It is intended that the specification and 
examples be considered as exemplary only, with a true scope and spirit of the Invention being indicated by 
the following claims. 



60 Claims 

1. In a distributed processing system containing a plurality of interconnected nodes including a clerk 
node and a plurality of server nodes providing time to the system, wherein the clerk node includes a local 
clock having a predetermined resolution value, keeping a local time, and having an inaccuracy associated 
55 therewith representing an amount by which that local time deviates from a conrect time value, a method for 
maintaining a correct time in the clerk node comprising the steps, performed by the clerk node, of: 
requesting an updated time interval from at least one of the server nodes when the local inaccuracy of the 
local clock of the clerk node exceeds a predetermined maximum inaccuracy value; 
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noting a request time according to the local clock of the clerk node when the clerk node requests said 
updated time interval; 

receiving from the sen/er nodes respective updated time interval representations and corresponding delay 

values; 

noting, for each receipt of said updated time inten/al representations and conresponding delay values, a 
different response time according to the local clock of the clerk node; 

calculating a correct time interval from said received updated time Interval representations, said received 
delay values, said noted request time, said noted response times, and the resolution value of the local clock 
of the clerk node, wherein the correct time Is contained within said calculated correct time interval; and 
adjusting the local time kept by the local clock of the clerk node according to said calculated correct time 
interval. 

2. The method of claim 1 . further including the step of: 

incrementing the local time of the local clock of the clerk node by periodically adding an Incrementation 
value equal to the predetermined resolution value to the local time kept by the local clock, and 
75 wherein said adjusting step further includes the step of: 

changing said Incrementation value according to said calculated correct time interval. 

3. The method of claim 2. wherein said adjusting step further includes the steps of: 

calculating a duration value to determine a length of time during which said changing step is performed: 
and 

20 restoring said incrementation value to the predetermined resolution value when said incrementation value 
changing step has been performed for a length of time equal to said calculated duration value. 

4. The method of claim 1, further including the step of: 

calculating a resynchronization time at which the inaccuracy of the local clock of the clerk node exceeds 
said predetermined maximum inaccuracy value, and 

wherein said requesting step is performed when the local clock of the clerk node has a time substantially 
equal to said calculated resynchronization time. 

5. The method of claim 4. further including the step of requesting an updated time interval from the 
server nodes when the local clock has a time substantially equal to a predetemnined maximum resynch- 
ronization time. , , w 

6. The method of claim 1, wherein each server node includes a local clock keeping a local lime and 
having an inaccuracy value associated therewith, and wherein the server nodes include time maintainer 
node which updates its local clock by said requesting, request time noting, receiving, response time noting, 
calculating, and adjusting steps used by the clerk, node and a time provider node which updates its local 
clock by accessing an external time source, wherein said requesting step further includes the steps. 

35 performed by the clerk node, of: 

requesting an updated time interval from the time maintainer node using a predetermined protocol; and 
requesting an updated time interval from a time provider node using said predetermined protocol. 

7. The method of claim 1 . wherein each server node includes a local clock keeping a local time and 
having an inaccuracy value associated therewith, and wherein the server nodes include a time maintainer 

40 node which updates its local clock by and requesting, request time noting, receiving, response time noting, 
calculating, and adjusting steps used by the clerk node and a time provider node which updates Its local 
clock by accessing an extemal time source, wherein said receiving step further includes the steps, 
performed by said clerk node, of: 

receiving an updated time interval representation and a corresponding delay value from the time maintainer 
45 node using a predetermined protocol; and 

receiving an updated time interval representation and corresponding delay values from the time provider 
node using said predetermined protocol. 

8. In a distributed processing system containing a plurality of interconnected server nodes, one of which 
is called a time maintainer node, wherein the server nodes provide time to the system and wherein the time 

50 maintainer node includes a local clock having a predetermined resolution value, keeping a local time, and 
having an inaccuracy associated therewith representing an amount by which that local time deviates from a 
con-ect time value, a method for maintaining a correct time in the time maintainer node comprising the 
steps, performed by the time maintainer node, ot 

requesting an updated time interval from at least one of the server nodes when the local inaccuracy of the 
56 local clock of the time maintainer node exceeds a predetermined maximum inaccuracy value; 

noting a request time according to the local clock of the time maintainer node when the time maintainer 
node requests said updated time interval; 

receiving from the sen/er nodes respective updated time interval representations and conresponding delay 
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values; 

noting, for each receipt of said updated time interval representations and corresponding delay values, a 
different response time according to the local clock of the time maintainor node; 

calculating a correct time interval from said received updated time interval representations, said received 
5 delay values, said noted request time, said noted response times, and the resolution value of the local dock 
of the time maintainer node, wherein the correct time is contained within said calculated correct time 
interval; and 

adjusting the local time kept by the local clock of the time maintainer node according to said calculated 
correct time interval. 
10 9. The method of claim 8, further including the step of: 

incrementing the local time of the local clock of the time maintainer node by periodically adding an 
incrementation value equal to the predetermined resolution value to the local time kept by the local clock, 
and 

wherein said adjusting step further includes the step of: 
IS changing said incrementation value according to said calculated correct time interval. 

10. The method of claim 9, wherein said adjusting step further includes the steps of: 

calculating a duration value to determine a length of time during which said changing step is performed; 
and 

restoring said incrementation value to the predetermined resolution value when said incrementation value 
20 changing step has been performed for a length of time equal to the calculated duration value. 

11. The method of claim 8. further including the step of: 

calculating a resynchronlzation time at which the inaccuracy of the local clock of the time maintainer node 

exceeds said predetermined maximum inaccuracy value, and 

wherein said requesting step is performed when the local clock of the time maintainer node has a time 
25 substantially equal to said calculated resynchronlzation time. 

1 2. The method of claim 8. further Including the step of requesting an updated time interval from the 
server nodes when the local clock has a time substantially equal to a predetermined maximum resynch- 
ronlzation time. 

1 3. The method of claim 8, further including the step of detecting as faulty server nodes those server 
30 nodes from which updated time interval representations outside of said calculated correct time interval are 

received. 

14. The method of claim 8. wherein each server node includes a local clock keeping a local time and 
having an inaccuracy value associated therewith, and wherein the server nodes include a second time 
maintainer node and a time provider node which updates Its local clock by accessing an external time 

35 source, wherein said requesting step further includes the steps, performed by the time maintainer node, of: 
requesting an updated time interval from the second time maintainer node using a predetermined protocol; 
and 

requesting an updated time interval from the time provider node using said predetermined protocol. 

15. The method of claim 8. wherein each server node includes a local clock keeping a local time and 
40 having an inaccuracy value associated therewith, and wherein the server nodes include a second time 

maintainer node and a time provider node which updates its local clock by accessing an external time 
source, wherein said receiving step further includes the steps, performed by the time maintainer node of, of: 
receiving an updated time interval representation and a corresponding delay value from the second time 
maintainer node using a predetermined protocol; and 
45 receiving an updated time interval representation and corresponding delay values from the time provider 
node using said predetermined protocol. 

16. The method of claim 8. wherein each of the server nodes possesses one of a plurality of epoch 
numbers identifying a group of nodes, 

wherein said time interval representation and delay value receiving step further Includes the step of 
50 receiving from the server nodes the epoch numbers of the server nodes, and 

further including the step of ignoring said updated time interval representations and corresponding delay 
values received from a server node when said received epoch number received from the server node 
differs from said epoch number of the time maintainer node. 

17. In a distributed processing system containing interconnected nodes including a server node and 
55 one other node, wherein the nodes include a local clock having a predetermined resolution value, keeping a 

local time, and having an inaccuracy associated therewith representing an amount by which that local time 
deviates from a coaect time value, a method for maintaining a correct time in the other node comprising the 
steps, performed by the server node, of: 
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receiving a request for an updated time interval from the other node wlien the local inaccuracy of the local 
clock of the other node exceeds a predetermined maximum inaccuracy value; 

noting a receipt time according to the local clock of the server node when the server node receives said 
request from the other node: 
5 calculating a delay value from said noted receipt time; 

calculating an updated time interval representation from the local time of the local clock of the server node 
and the inaccuracy of the local clock of the server node; and 

sending said updated time interval representation and said corresponding delay value to the other node. 

18. The method of claim 17. wherein the sen/er node possesses one of a plurality of epoch numbers 
ro identifying a group of nodes, and 

wherein said sending step further includes the step of sending the epoch number of the server node to the 

other node. ^ • , ^- i 

19, In a distributed processing system containing a plurality of interconnected nodes including a clerk 
node and a server node providing time to the system, wherein each of the nodes includes a local clock 
having a predetermined resolution value, keeping a local time, and having an inaccuracy associated 
therewith representing an amount by which that local time deviates from a correct time value, a method for 
maintaining a correct time in the clerk node comprising the steps of: 

requesting, by the clerk node, an updated time interval from the server node when the local inaccuracy of 
the local clock of the clerk node exceeds a predetermined maximum inaccuracy value; 
noting, by the clerk node, a request time according to the local clock of the clerk node when the clerk node 
requests said updated time interval; 

receiving, by the clerk node, from the server node an updated time interval representation and con-espond- 

ing delay value; 

noting, by the clerk node, for said receipt of said updated time interval representation and corresponding 
25 delay value, a response time according to the local clock of the clerk node; 

calculating, by the clerk node, a correct time interval from said received updated time interval representa- 
tion, said received delay value, said noted request time, said noted response time, and the resolution value 
of the local clock of the clerk node, wherein the correct time is contained within said calculated correct time 
interval; 
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adjusting, by the clerk node, the local time kept by the local clock of the clerk node according to said 
calculated correct time interval; 

receiving, by the server node, said request for an updated time interval from the clerk node; 
noting, by the sen/er node, a receipt time according to the local clock of the server node when the server 
36 node receives said request from the clerk node; 

calculating, by the server node, a delay value from said noted receipt time; 

calculating, by tiie server node, an updated time Interval representation from the local time of the local 
clock of the server node and the inaccuracy of the local clock of the server node; and 

sending, by the server node, said updated time interval representation and said corresponding delay value 
40 to the clerk node. 

20. The method of claim 19. wherein the server node possesses one of a plurality of epoch numbers 
identifying a group of nodes, and 

wherein said sending step further includes the step, performed by the server node, of sending the epoch 
number of the server node to the clerk node. 
45 21. The method of claim 19. further including the step, performed by the clerk node, of: 

Incrementing the local time of the local clock of the clerk node by periodically adding an Incrementation 
value equal to the predetermined resolution value to tiie local time kept by the local clock of the clerk node, 
and 

wherein said adjusting step further Includes the step, performed by the clerk node, of changing said 
so incrementation value according to said calculated correct time interval. 

22. The method of claim 21, wherein said adjusting step further includes the steps, perfomned by the 

clerk node, of: " . . ^ ^ 

calculating a duration value to determine a length of time during which said changing step is performea; 

and 

55 restoring said Incrementation value to the predetermined resolution value when said changing step has 
been performed for a length of time equal to said calculated duration value. 

23. The metiiod of claim 19. furttier including the step, perfomned by the clerk node, of calculating a 
resynchronization time at which the Inaccuracy of the local clock of the clerk node exceeds said 
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predetermined maximum inaccuracy value, and 

wherein said requesting step is performed by the clerk node wlien the local clock of the clerk node has a 
time substantially equal to said calculated resynchronization time. 

24. The method of claim 23, further Including the step, performed by the clerk node, of requesting an 
5 updated time interval from the server node when the local clock of the clerk node has a time substantially 

equal to a predetermined maximum resynchronization time. 

25. The method of claim 19. wherein the distributed processing further contains a plurality of 
interconnected server nodes including a time maintainor node which updates its local clock by said 
requesting, request time noting, receiving, response time noting, calculating, and adjusting steps used by 

70 the clerk node and a time provider node which updates its local clock by accessing an external time source, 
wherein said requesting step performed by the clerk node further includes the steps of: 
requesting an updated time interval from the time maintainor node using a predetermined protocol; and 
requesting an updated time interval from the time provider node using said predetermined protocol. 

26. The method of claim 19, wherein the distributed data processing system further contains a plurality 
IS of interconnected server nodes including a time maintainer node which updates its local clock by said 

requesting, request time noting, receiving, response time noting, calculating, and adjusting steps used by 
the clerk node and a time provider node which updates its local clock by accessing an external time source, 
wherein said receiving step performed by the clerk node further includes the steps of: 

receiving an updated time interval representation and a corresponding delay value from the time maintainer 
20 node using a predetermined protocol: and 

receiving an updated time interval representation and a corresponding delay value from the time provider 
node using said predetermined protocol. 

27. The method of claim 19. wherein the server node is a time maintainer node, wherein the distributed 
processing system further contains a second server node providing time to the system, and further 

25 comprising the steps, performed by the time maintainer node, of: 

requesting an updated time Interval from the other server node when the local inaccuracy of the local clock 
of the time maintainer node exceeds a predetermined maximum inaccuracy value; 

noting a request time according to the local clock of the time maintainer node when the time maintainer 
node requests said updated time interval: 
30 receiving from the other server node an updated time interval representation and a corresponding delay 

value; 

noting, for said receipt of said updated time Interval representation and corresponding delay value, a 
response time according to the local clock of the time maintainer node; 

calculating a connect time interval from said received updated time interval representation, said received 
35 delay value, said noted request time, said noted response time, and the resolution value of the local clock 
of the time maintainer node, wherein the correct time is contained within said calculated correct time 
interval: and 

adjusting the local time kept by the local clock of the time maintainer node according to said calculated 
correct time interval. 

40 28. The method of claim 27. further including the step, performed by the time maintainer node, of: 

incrementing the local time of the local clock of the time maintainer node by periodically adding an 
incrementation value equal to the predetermined resolution value to the local time kept by the local clock, 
and 

wherein said adjusting step performed by the time maintainer node further includes the step of changing 
45 said incrementation value according to said calculated correct time interval. 

29. The method of claim 28, wherein said adjusting step performed by the time maintainer node further 
includes the steps of: 

calculating a duration value to determine a length of time during which said changing step is performed; 
and 

so restoring said incrementation value to the predetermined resolution value when said incrementation value 
changing step has been, performed for a length of time equal to the calculated duration value. 

30. The method of claim 27, further including the step, performed by the time maintainer node, of: 
calculating a resynchronization time at which the inaccuracy of the local clock of the time maintainer node 
exceeds said predetermined maximum Inaccuracy value, and 

55 wherein said requesting step is performed by the time maintainer node when the local clock of the time 
maintainer node has a time substantially equal to said calculated resynchronization time. 

31. The method of claim 27. further including tiie step, performed by the time maintainer node, of 
requesting an updated time interval from the other server node when the local clock has a time substantially 
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laniiai to a nredetermined maximum resynchronization time. 

32 The memod of claim 27. further including the step, performed by the time ma.nta.ner node of 
detec'ng as TauTtJ server nodes those server nodes from which updated time interval representations 

outside of said calculated correct time interval are received. 

outs^e of ca ^^^^^.^ ^^^^^ ^^^^^ ^^^^ ^ ^.^^ p^^^,^^, ^j,,^^ ""'"t" '2 

locarclock ^ accessing an external time source, wherein the distributed process.ng ^Y^terr, further 
contlns a second time maintainer node providing time to the system, and where.n sa.d requesfng step 

oerformed bv the time maintainer node further includes the steps of: ^ . • ^ 

SquesTng an updated time interval from the second time maintainer node using a predeterm.ned protocol. 

Sauesting an updated time interval from the time provider node using said predetermined P[oto<=°': 
request.ng an up ^^^^^^^ ^^^^^ ^^^^^ ^ ^.^^ p^^^,^^^ wh.ch updates .te 

.ocarclock by accessing an external time source, wherein the distributed processing sv-tem further 
com^ns a second time maintainer node providing time to the system, and where.n sa.d requestng step 

TeS^rpdS r rr::rpr=ir^d corresponding delay values from the time provider 

^ ''^\fZ'ZZT^:::T:^::^n of *e serv. nodes possesses one of a plurality of epoch 

'ZZ'Xlis^^^^^^ and delay value receiving step performed by the «me m Jnt.-ner 

node lurth^r includes the step of receiving from the other server nodes the epoch numbers of the server 



25 



30 



35 



i!!?h«; frldtna the step of ignoring said updated time interval representations and corresponding delay 
lats ;rceTved% 'aid received epoch number received from said sender node 

differs from said epoch number of the time maintainer node. 

36 The method of claim 19. wherein the sen/er node is a time provider node wh.ch updates ts local 
clocfby accestrng an external time source, wherein the distributed processing ^Y^^"" 

server node providing time to the system, and further Including the steps, performed by the t.me 

provider node, of: 

reauesting an updated time interval from the other server node periodically: 
notirlg a rJque^ time according to the local clock of the time provider node when the t.me provider node 

;2eTrgt^m"^:*tthrsLr^^^ an updated time interval representation and corresponding delay 

noting, for said receipt of said updated time interval representation and conresponding delay value, a 
resDonse time according to the local clock of the time provider node; ,^^:„^h 
SS^g Tcorrect time Interval from said received updated time interval l^^^^^'^^'^^'l' ^^'^^^^^ 
deSv va"ue said noted request time, said noted response time, and the resolution value of me local ctack 
J me tirle prSIkler node, wherein the correct time is contained within said calculated conrect tme interval. 

Setcting as faulty server nodes those server nodes from which updated time interval representations 
niit<!ide of said calculated conrect time interval are received. 

37 A cTerk no5e in a distributed processing system containing a plurality of interconnected server 
nodes.' the clerk node providing a correct time to an application program in the system and composing: 
a memory having a location accessible by the application program; 

a local clock keeping a local time having an associated inaccuracy; .„^i^,,u.. 
rncrS.enting means connected to said local clock for causing said local time to increase Pe"OdK:alW; 
S wSesSng means connected to at least one server node and responsive to a first senes of 'nsmictions 
sSedTn sld merory..^ tiiat said local time of said local clock is within a predetermined 

inaccuracv of the correct time, said processing means including: ^ . ^* 

met^s^r requiting an upKlated time interval from at least one of the server nodes when tiie inaccuracy of 

said local clock exceeds a predetermined majdmum inaccuracy value. sh ..nrfateri 

rn^nl for noting a request time according to said local clock when the clerk node requests said updated 

SSarisT'^eceiving from ti« server nodes respective updated time internal representations and cor- 
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responding delay values. 

means for noting, for each receipt of said updated time interval representations and corresponding delay 
values, a different response time according to said local clock, 

means for calculating a correct time interval from said received updated time interval representations, said 
5 received delay values, said noted request time, said noted response times, and said resolution value of said 
local clock, wherein the correct time is contained within said calculated correct time interval, and 
means for adjusting said local time kept by said local clock according to said calculated correct time 
interval; and 

second processing means, connected to said memory and to said local clock and responsive to a second 
10 series of instructions stored in said memory, for storing said local time kept by said local clock into the 
memory location accessible by the application program. 

38. A time maintainer node In a distributed processing system containing a plurality of interconnected 
server nodes providing time information to the system, the time maintainer node maintaining a correct time 
and comprising: 

75 a memory: 

a local clock keeping a local time having an associated inaccuracy; 

incrementing circuit means connected to said local clock for causing said local time of said local clock to 
increase periodically; and 

processing means connected to at least one server node and responsive to a series of instructions stored in 
20 said memory, for ensuring that said local time of said local clock is within a predetermined inaccuracy of 
the correct time, said processing means Including: 

means for requesting an updated time interval from at least one of the server nodes when the inaccuracy of 
said local clock exceeds said predetermined maximum inaccuracy value, 

means for noting a request time according to said local clock when the time maintainer node requests said 
25 updated time interval, 

means for receiving from the server nodes respective updated time interval representations and cor- 
responding delay values. 

means for noting, for each receipt of said updated time interval representations and corresponding delay 
values, a different response time according to said local clock, 
30 means for calculating a correct time interval from said received updated time inrerval representations, said 
received delay values, said noted request time, said noted response times, and said resolution value of said 
local clock, wherein the correct time is contained within said calculated correct time interval, and 
means for adjusting the local time kept by said local clock according to said calculated correct time interval. 

39. The time maintainer node of claim 38. wherein tiie distributed processing system furtiier includes a 
35 requesting node that requests an updated time interval, 

and wherein the time maintainer node further includes: 

second processing means, connected to the requesting node and to said local clock, responsive to a 
second series of instructions stored in said memory, for responding to a request from the requesting node, 
said second processing means including: 
40 means for receiving the request for an updated time interval from the requesting node, 

means for noting a receipt time according to said local clock when the time maintainer node receives the 
request from the requesting node, 

means for calculating a delay value from said noted receipt time. 

means for calculating an updated time interval representation from the local time of said local clock and tiie 
45 inaccuracy of said local clock, and 

means for sending said updated time Interval representation and said corresponding delay value to tiie 
requesting node. 

40. The time maintainer node of claim 39, wherein said time maintainer node possesses one of a 
plurality of epoch numbers identifying a group of nodes, and 

50 wherein said second processing means furtiier includes means for sending tiie epoch number of the time 
maintainer node to tiie requesting node. 

41. A time provider node in a distributed processing system containing a plurality of interconnected 
server nodes, the time provider node maintaining a correct time and comprising: 

a memory; 

55 a local clock keeping a local time having an associated inaccuracy; 

Increment circuit means connected to said local clock for causing said local time to Increase periodically; 
external time source Interface means for receiving the conrect time from an external time source; and 
processing means connected to said external time source Interface means and executing instructions stored 
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in the memory for adjusting the local lime kept by said local clock according to said received correct time. 

42. The time provider node of claim 41, wherein the distributed processing system further includes a 
requesting node that requests an updated time inten^al, wherein the time provider node further includes: 
second processing means connected to said local clock and to the requesting node, responsive to a second 
5 series of instructions stored in said memory, for responding to the request for an updated time interval, said 
second processing means including: 

means for receiving the request for an updated time interval from the requesting node, 
means for noting a receipt time according to said local clock when the time provider node receives said 
request from the requesting node. 
10 means for calculating a delay value from said noted receipt time. 

means for calculating an updated time Interval representation from the local time of said local clock and the 
inaccuracy of the local clock, and , *u 

means for sending said updated time interval representation and said corresponding delay value to the 

requesting node. , * 

IS 43. The time provider node of claim 42, wherein the time provider node possesses one of a plurality of 

epoch numbers identifying a group of nodes, and 

wherein said second processing means further includes means for sending the epoch number of the time 
provider node to the requesting node. 
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